ADFA-4739: Pipeline for producing template-based Kotlin documentation - #21
ADFA-4739: Pipeline for producing template-based Kotlin documentation#21alexmmiller wants to merge 16 commits into
Conversation
configJson was a dedicated Json instance used only for the manual config-decode fallback path. Inlining it removes the standalone property while keeping the same ignoreUnknownKeys tolerance for extra keys in user-authored plugin config. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
- scripts/verify_package_index.py: confirms every object listed in a package's index.json actually has documented content on the page its url points to (matching by dri), not just that the file exists. - scripts/kotlin/test_kotlin_stdlib.sh: pure-bash check that the default HTML build and the kdoc-to-json JSON build have a one-to-one set of pages (extension-swap aware, in both directions). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…, generate and add templates to display it to database, optimize and insert Kotlin website media. Script to sync current kotlin-stdlib documentation against a newly-generated documentation set (for now, used to do pruning for ADFA-4737 https://appdevforall.atlassian.net/browse/ADFA-4737)
|
Claude says: Issues found
per ADFA-4737, only the .py files from the original working directory were copied here. templates/ and assets/ ... must be placed alongside these scripts before running populate_db.py. But the PR itself does include templates/{page.peb,nav.peb,nav.html}, assets/{docs.css,tabs.js,sidebar.js}, and config.json (added by the "Config/assets for pipeline test" commit). A reviewer or new user following the README literally would go hunting for files that are already sitting right there. Worth updating that note (or clarifying it only applied to the original copy-over before the pipeline-test commit added them).
python This is clearly a leftover from the author's local machine. It's harmless in that a wrong path just fails the os.path.isfile check, but it shouldn't ship as the default in a shared repo — either make it required (no default) or default to documentation.db in cwd, matching populate_db.py's convention.
python If one rename's new_name happens to equal another rename's old_name (e.g. foo.png → foo.webp from one optimization, and some unrelated foo.webp → foo-2.webp renamed elsewhere), the second substitution could re-match text that the first substitution just wrote, double-rewriting a reference that was never actually pointing at the second file. It's a narrow edge case (requires two renames whose old/new basenames chain), but since build_rename_map only guards against the same old name mapping to two different new names, not this chain scenario, it's worth at least a comment acknowledging the assumption (or building a single combined regex/dict-based single-pass substitution instead of N sequential replaces).
|
- README: templates/assets are already included in the repo, not left for the user to place separately. - sync_kdoc_json_to_db.py: default --db to documentation.db in cwd instead of a personal /home/alex path; add explicit BEGIN to match sibling scripts' transaction style. - insert_optimized_media.py: rewrite_pages now substitutes renamed image references in a single regex pass over each row's original text, instead of sequential str.replace calls on a mutating buffer, which could double-rewrite a reference if one rename's new name chained into another rename's old name. - run_e2e_pipeline_test.sh: add a verification step that reuses populate_db.py's own pruning logic to confirm blacklisted topic pages are actually absent from the database, replacing a weaker proxy check that only tested for k/html/wasm% paths. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
|
e2e_run_output.log Summary: == Summary == == Blacklist pruning verification == Done. Backups (populate_db.py, insert_optimized_media.py, and sync_kdoc_json_to_db.py |
Brings in the updated KDoc-to-JSON Dokka plugin (sourceSet whitelisting, package-index/structure verification scripts) so this branch has the current plugin used to generate the kotlin-stdlib docs consumed by sync_kdoc_json_to_db.py.
- run_e2e_pipeline_test.sh: replace the manually-supplied STDLIB_ALL_LIBS
path with STDLIB_DOCS_DIR (a kotlin repo's libraries/tools/kotlin-stdlib-docs
checkout). New Step 4/5 derives the repo root and calls
build-stdlib-json-docs.sh to freshly build/publish the kdoc-to-json plugin
and generate kotlin-stdlib/-reflect/-test JSON docs (common+jvm source
sets only, per the existing plugin config) before Step 5/5 syncs them into
the database.
- Add build-stdlib-json-docs.sh: builds a fresh copy of the plugin, swaps in
the JSON-plugin-enabled build.gradle.kts for the duration of the build
(restored on exit via trap), and prints only the resulting all-libs path
to stdout so it composes as STDLIB_ALL_LIBS="$(build-stdlib-json-docs.sh
<kotlin-repo-root>)" - every gradlew invocation inside it is redirected to
stderr so that capture isn't polluted by build console output.
- build.gradle.kts: re-add the Dokka dev-snapshot Maven repo (same
dokka_repository property/default kotlin-stdlib-docs' own
settings.gradle.kts uses) - kotlin-stdlib-docs' own
dokka-samples-transformer-plugin subproject needs a Dokka dev build that
isn't on Maven Central, and the allprojects{} repositories block here had
removed access to it for every subproject.
Verified end-to-end against a real kotlin checkout: fresh plugin build,
JSON generation (common+jvm only), and sync into a scratch documentation.db
all succeed, with blacklist pruning verification passing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Summary
An end-to-end pipeline for producing templated Kotlin documentation.
Changes
Kotlin website docs
kotlin-web-site/docs(Writerside Markdown) into JSON, build nav, optimize media, and insert everything intodocumentation.db, intoProcessDocs/ProcessKotlinDocs/ProcessKotlinWebsiteJSON/. Adds a README documenting each script's purpose, required inputs (kr.tree,webHelpImages.zip, config JSON), and end-to-end usage — includingpopulate_db.py --blacklisted-element-titles, the mechanism this ticket needs for pruning docs sections (e.g. Kotlin/Wasm) we don't want to ship.Kotlin-stdlib docs
sync_kdoc_json_to_db.pyscript into a new directoryscripts/sync_kotlin_stdlib_docs/, for syncing generated kotlin-stdlib docs against the database.Templated docs pipeline
ProcessDocs/ProcessKotlinDocs/run_e2e_pipeline_test.shwhich builds the kotlin-stdlib and Kotlin website JSON documentation and inserts it into CoGo's documentation database.Test plan for Kotlin website docs:
/home/alex/ADFA/KotlinWebsiteProcess/andDokka-plugin-kdoc2json/scripts/(no edits were made during the copy)populate_db.pywith--blacklisted-element-titlesagainst a scratch databaseNote on KDoc-to-JSON Dokka plugin version
We currently include a stale copy of the KDoc-to-JSON Dokka plugin while that work is still being done this PR. Once those changes are merged into
mainwe'll merge them into this branch.